Intelligent traffic steering over optimal paths using multiple access technologies

ABSTRACT

One or more processors in a communication system allocate a selected connectivity path from a plurality of connectivity paths supported by a plurality of access technologies. The selected connectivity path is allocated to a service flow between a communication device and a core network. One or more available access channels are assigned to the service flow for the selected connectivity path. Network conditions and performance of the service flow are monitored on the one or more available access channels. The selected connectivity path is modified based on the network conditions and the performance of the service flow seamlessly to the existing applications or service flows. In some cases, criteria for modifying the selected connectivity path are modified for a predetermined time interval in response to modifying the selected connectivity path, e.g., by modifying threshold values that are used to determine whether to modify the selected connectivity path based on a comparison of the threshold values to the network conditions or the performance of the service flow.

BACKGROUND

Communication systems support network connectivity via wired or wirelessconnections and current trends favor fixed-wireless convergence tosupport unified access across wired and wireless connections.Communication devices such as smart phones, tablets, and laptops areable to maintain concurrent connections to a network via multiple accesstechnologies such as cellular technologies that operate in a licensedfrequency band and other technologies that operate in unlicensedfrequency bands. For example, user equipment can implement multipleaccess technologies according to Third Generation (3G) standards definedby the Third Generation Partnership Project (3GPP) such as Long-TermEvolution (LTE), Fourth Generation (4G) standards, Fifth Generation (5G)standards, 802.11 standards defined by IEEE such as Wi-Fi, or usingother access technologies such as millimeter wave technologies. Someuser equipment also support communication over wired connections such asdigital subscriber line (DSL) connections, which can be usedconcurrently with wireless connections. Other types of communicationdevices such as devices that support machine-to-machine communicationare also able to support multiple access technologies that can be usedto maintain concurrent wireless connections.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be better understood, and its numerousfeatures and advantages made apparent to those skilled in the art byreferencing the accompanying drawings. The use of the same referencesymbols in different drawings indicates similar or identical items.

FIG. 1 is a block diagram of a first example of a communication systemaccording to some embodiments.

FIG. 2 is a block diagram of a second example of a communication systemthat implements intelligent path selection according to someembodiments.

FIG. 3 is a block diagram of a network function virtualization (NFV)architecture according to some embodiments.

FIG. 4 is a flow diagram of a first portion of a method of selecting aninitial path for a new service flow in a communication system thatsupports multiple access technologies according to some embodiments.

FIG. 5 is a flow diagram of a second portion of a method of selecting aninitial path for a new service flow in a communication system thatsupports multiple access technologies according to some embodiments.

FIG. 6 is a flow diagram of a method of performance monitoring anddynamic path switching in a communication system that supports multipleaccess technologies according to some embodiments.

FIG. 7 is a flow diagram of a first portion of a method of dynamicallymodifying criteria for performing path reselection or path consolidationin a communication system that supports multiple paths using multipleaccess technologies according to some embodiments.

FIG. 8 is a flow diagram of a second portion of a method of dynamicallymodifying criteria for performing path reselection or path consolidationin a communication system that supports multiple paths using multipleaccess technologies according to some embodiments.

FIG. 9 is a flow diagram of a third portion of a method of dynamicallymodifying criteria for performing path reselection or path consolidationin a communication system that supports multiple paths using multipleaccess technologies according to some embodiments.

FIG. 10 is a flow diagram of a fourth portion of a method of dynamicallymodifying criteria for performing path reselection or path consolidationin a communication system that supports multiple paths using multipleaccess technologies according to some embodiments.

FIG. 11 is a flow diagram of a fifth portion of a method of dynamicallymodifying criteria for performing path reselection or path consolidationin a communication system that supports multiple paths using multipleaccess technologies according to some embodiments.

FIG. 12 illustrates results that indicate numbers of active userequipment that are connected to LTE or Wi-Fi according to someembodiments.

FIG. 13 illustrates results that demonstrate reliability of intelligenttraffic steering according to some embodiments.

FIG. 14 illustrates a cumulative distribution function (CDF) of thethroughput for an FTP download according to some embodiments.

FIG. 15 illustrates a CDF of the throughput for an FTP download timewith and without intelligent traffic steering according to someembodiments.

FIG. 16 illustrates a CDF of packet delays for an Internet gamingapplication according to some embodiments.

DETAILED DESCRIPTION

Conventional communication systems implement different techniques toleverage the availability of multiple access technologies to provide animproved user experience. Systems that operate according to 3G/4G andWi-Fi implement handover to support user mobility. For example, userequipment can be handed over from an LTE connection to a Wi-Ficonnection in response to the user equipment detecting deterioratingsignal quality on the LTE connection if an interworking mechanism issupported by the mobile network operator. Even if connectivity could bemaintained using the handover mechanism, it requires disconnecting acommunication session with the LTE connection and forming a newconnection using the Wi-Fi connection, which may disrupt ongoing flowsbetween the network and the user equipment. The primary goal of thehandover mechanism is to maintain user's connectivity under mobilityenvironment. Handover is not used to support optimal networkconnectivity, which aims at providing the highest quality of experience,including desirable throughput, low packet delay, and reliableconnectivity.

Splitting flows across multiple interfaces provides another way toenhance user experience using multiple access technologies. For example,Hypertext Transfer Protocol (HTTP) range requests can be used to dividefiles for transmission over different network links and migrate datasessions from one link to another. However, this approach only workswith HTTP applications and does not support other applications based onFile Transfer Protocol (FTP), User Datagram Protocol (UDP), or HTTPSecure (HTTPS). For another example, multipath transmission controlprotocol (MPTCP) aggregates multiple TCP subflows that are conveyed overdifferent network paths to the same end device. However, not allendpoints are MPTCP capable. Furthermore, even if an MPTCP proxy is usedin the network and the user device supports the MPTCP protocol, the userpackets bypass the proxy if the TCP packets are encrypted using IPSecbecause the TCP header information in the encrypted packet is invisibleto the proxy. Multipath TCP does not support UDP traffic. For yetanother example, IP packets could combine multiple network interfacesusing network interface bonding. However, this approach may not supportpacket reordering mechanism if packets arrive out-of-order. Even thoughthe IEEE 802.3ad standard specifies packet in-order delivery mode, itcomes with a drawback, mandating all devices in the aggregate operate atthe same speed and duplex. Furthermore, scaling up IP packet aggregationis difficult because new interfaces and hardware have to be provisionedlocally, which can worsen the out-of-order packet delivery problem. Thekey objective for the aforementioned techniques is to provide higherthroughput using throughput aggregation, but these approaches do notaccount for quality requirements of delay sensitive services.

FIGS. 1-16 disclose a heuristic, non-greedy approach to providingindividual service flows (or subflows) terminated by wirelesscommunication devices that support multiple access technologies withnearly proportionally fair resource allocations, while also satisfyingper-flow quality-of-service (QOS) requirements. In some embodiments, oneof a plurality of paths supported by a corresponding plurality of accesstechnologies is selectively allocated to a flow (or subflow) based on atleast one of an operator policy or a QOS requirement for the flow. Forexample, the flow can be allocated according to an operator policy,bandwidth availability, measured packet delays, or jitter associatedwith the plurality of paths. Once the path is allocated, availableaccess channels are assigned to the flow. Performance of the flow on theassigned channel is monitored. In some embodiments, network conditionsare also monitored. Path reselection is performed to modify theallocated path based on the monitored performance of the flow and thenetwork conditions. For example, the flow can be switched to a differentpath associated with a different access technology in response to aperformance metric falling below a threshold value. Flows that have beensplit across multiple access channels can be consolidated into a singleflow.

Switching a flow from a first path that operates according to a firstaccess technology to a second path that operates according to a secondaccess technology does not necessarily immediately improve theperformance of the flow. For example, switching a flow from a relativelyslow network connection to a relatively fast network connection toachieve a target throughput may not immediately increase the throughputof the flow to the target throughput. Depending on the transport layerprotocols used by the fast and slow networks, the timescale forincreasing throughput along the newly selected high-speed networkconnectivity path may be longer than an estimated trigger time forperforming a path switch. This may lead to a ping-ponging of the flowbetween the first and second paths during the ramping up time intervalif the throughput along the newly selected path does not reach anexpected target performance metric (e.g., a target throughput) withinthe trigger time.

Ping-ponging is avoided by monitoring progress towards the targetperformance metric, such as the target throughput. In some embodiments,criteria such as threshold values of the target performance metric aredynamically modified in response to path reselection. For example, adynamic target throughput can be set to a relatively low value inresponse to the path reselection and then increased during a ramping uptime interval. As long as a measured average throughput exceeds thedynamic target throughput, the path reselection is maintained. A newpath reselection is performed in response to the performance metricfailing to achieve the dynamic threshold value of the target performancemetric. For example, a new path reselection is performed in response toa measured average throughput being less than a current value of thedynamic target throughput, which indicates that the previous pathreselection is not improving performance for the flow.

FIG. 1 is a block diagram of a first example of a communication system100 according to some embodiments. The communication system 100 includesbase stations 101, 102, 103 (collectively referred to herein as “thebase stations 101-103”) that operate according to different accesstechnologies. For example, the base station 101 operates according to4G, 5G, and Wi-Fi standards, the base station 102 operates according toWi-Fi standards, and the base station 103 supports millimeter wavecommunication. The base stations 101-103 provide wireless connectivityto one or more user equipment 105. The communication system 100 alsoincludes a gateway 110 that supports communication between the basestations 101-103 and a core network 115. The communication system 100also includes a cloud network 120.

The user equipment 105 implements an application layer 121 to supportapplication layer interfaces, as well as resource sharing and remotefile access. The user equipment 105 also implements a transport layer123 that supports reliable transmission of data segments betweencommunication endpoints including segmentation of packets,acknowledgment, and multiplexing. Some embodiments of the transportlayer 123 implement a transmission control protocol (TCP) or a userdatagram protocol (UDP). The user equipment 105 also implements anetwork layer such as an Internet protocol (IP) layer 125 that performsfunctions including addressing, routing, and traffic control for packettransmissions. The user equipment 105 further implements multiple accesstechnologies 130, 131, 132, which are collectively referred to herein as“the ATs 130-132.” For example, the AT 130 can be a cellular AT such as4G or 5G, the AT 131 can operate according to a standard such as Wi-Fiin the unlicensed frequency band, and the AT 132 can support millimeterwave communication. More or fewer ATs which can include wired accesstechnologies, are implemented in some embodiments of the user equipment105.

The gateway 110 implements a transport layer 135 such as a TCP enhancedlayer, an IP layer 136, and an L2/L1 layer 137 that supports reliabletransmission of data frames between nodes connected by physical layerand the transmission and reception of bit streams over a physicalmedium.

The communication system 100 supports multi-connectivity path selectionfor services provided to the user equipment 105. The illustratedembodiment depicts examples of network elements and interfaces involvedin the path selection algorithmic framework. The multi-connectivity pathselection service as discussed in the context of a user equipment 105that supports multimode connectivity and is in the same administrativedomain of multiple access technologies, such as the ATs 130-132. Thecommunication system 100 implements intelligent network controlledtraffic steering to enable the highest quality-of-experience (QoE) tousers while maintaining high network utilization, which is accomplishedby continuously finding and adapting the best connectivity path per userper service flow.

Some embodiments of the communication system 100 therefore implementnetwork elements that are configured to identify and assign the accesstechnology that supports at least the minimum service requirements(e.g., targeted throughput and/or packet delay) and that is the leastutilized among all the available access technologies. A linkselection/load balancing algorithm that operates on a longer timescaleensures that the mobile devices are connected to the best link inaverage. Some embodiments of the intelligent network steering algorithmimplemented in the communication system 100 use information that isactively measured (e.g., throughput and/or packet delay) whenever deemednecessary, along with radio conditions and radio resource measurementsthat are reported by the modules identified in this architecture, whichare described below.

A Radio Connection Manager (RCM) 140 is hosted in the base station 101,which implements multiple access technologies and therefore supportstight integration of the multiple access technologies (e.g., 4G, 5G,Wi-Fi). The RCM 140 is responsible for decision making and control andis a possible placement for some embodiments of the intelligent networksteering techniques disclosed herein. For example, the RCM 140 can beconfigured to support initial path selection of new incoming serviceflows as well as for intelligent traffic steering of existing serviceflows.

A Multi-Path Manager (MPM) 141 is implemented in the cloud network 120.Due to its placement in the network, which is deeper in the cloudcompared to RCM 140, the MPM 141 has a broader network control span incomparison to the RCM 140, above and beyond the radio path (e.g., thenetwork control span of the MPM 141 may extend control beyond radioaccess to include the transport segments of the network. For instance,the MPM 141 may reside in a Software Defined Network (SDN) controller,which enables such extended network controlled scope.

Multi-Link Schedulers (MLS) 142, 143 are hosted by the base station 101and the gateway 110, respectively. The MLS 142, 143 operate on user dataplane taking commands from controllers that are implemented in the RCM140 and MPM 141 to ensure separation between control and user planefunctions. The MLS 142, 143 are data plane modules that may or may notbe physically collocated with a corresponding control plane module. Inthe illustrated embodiment, the MLS 142 is physically co-located withthe RCM 140 and the MLS 143 is located in the gateway 110, while the MPM141 is physically separated from the gateway 110. The MLS 142, 143 aretriggered by the RCM 140 or the MPM 141 to perform user plane functions.For example, the MPM 141 or the RCM 140 selects a target accesstechnology based on algorithmic intelligence and triggers MLS 142, 143to establish a connectivity path through the selected access technology.The MLS 142, 143 measure network delay performance per mobile device andmeasures throughput per service flow along the user data routing pathbetween the user equipment 105 and the base station 101. In response toinstructions from the RCM 140 or the MPM 141, the MLS 142, 143 switchuser data paths among the accessible access technologies. The MLS 142,143 then provide user data packet in-order sequencing and removesduplicated packets.

Resource Measurement Managers (RMMs) 144, 145, 146 (collectivelyreferred to herein as “the RMMs 144-146”) are implemented in the basestations 101-103, respectively. The RMMs 144-146 are configured tomonitor radio resource utilization of a specific access technology andreports the measurements to the MPM 141 and the RCM 140. The RMMs144-146 are also configured to monitor uplink user data throughput peruser per service flow over a specific access technology, and reports itto the MPM 141 and the RCM 140. The RMMs 144-146 are further configuredto monitor uplink and downlink radio conditions per mobile device forall mobile devices connected to a specific access technology, which caninclude the user equipment 105. Information indicating the radioconditions includes a received signal strength indicator (RSSI), asignal to interference plus noise ratio (SINR), a modulation and codingscheme (MCS), and the like. The RMMs 144-146 configured to reportindications of the radio conditions to the MPM 141 and the RCM 140. TheRMMs 144-146 are also configured to generate and maintain statistics onthe number of connected users and their uplink/downlink traffic for aspecific access technology.

The user equipment 105 implements a User Connection Manager (UCM) 147,which is configured to establish a multi-access path selectionmanagement session to the MPM 141 or the RCM 140. The UCM 147 is alsoconfigured to monitor performance metrics such as downlink throughput ordelay measurements for the user equipment 105 and reports the measuredperformance metrics to the MPM 141 and the RCM 140. The UCM 147 alsoprovides downlink user data packet in-order sequencing and removesduplicate packets. The UCM 147 establishes an IP tunnel per accesstechnology between the user equipment 105 and the node hosting the MPM141 or the RCM 140. The UCM 147 also monitors the radio interfaceconditions (e.g., RSSI, SINR, MCS) for the user equipment 105 andtransmits measurement reports to the MPM 141 or the RCM 140 with anadjustable reporting frequency. For example, the UCM 147 can reportmeasurements periodically, in response to a request from the network, inresponse to a measured value of a performance network exceeding orfalling below a threshold, and the like. The UCM 147 steers the uplinkpath either based on decisions made by the MPM 141 or the RCM 140, orbased on a local decision taken at the UCM 147.

FIG. 1 also depicts conceptual network reference interfaces on thecontrol plane: Mr 150, Ur 151, Um 152, Rm_lte 153, Rm_wifi 154, and Rm_5g 155. The network interfaces 150-155 are used to communicate among thecorresponding network elements. In embodiments of the communicationsystem 100 that are implemented according to software definednetworking, the interfaces 150-155 act as control links between thevarious entities of a programmable network. In the illustratedembodiment, the MPM 141 is also connected to a wireless controller 160and a network controller 165.

Some embodiments of the communication system 100 implement a “best-rate”path selection algorithm for multi-connectivity networks with varioustypes of service flows such that the rates received by the variousservice flows under the “best-rate” path selection, in conjunction withlocal fair resource sharing at the individual access points, are closeto globally Proportional Fair. Some embodiments of the communicationsystem 100 therefore allocate a selected path from a plurality of pathssupported by multiple access technologies. The selected path isallocated to a flow between the user equipment 105 and the core network115. One or more available access channels are assigned to the flow forthe selected path. Network conditions and performance of the flow aremonitored on the one or more available access channels. The selectedpath is modified based on the network conditions and the performance ofthe flow. In some cases, criteria for modifying the selected path aremodified for a predetermined time interval in response to modifying theselected path, e.g., by modifying threshold values that are used todetermine whether to modify the selected path based on a comparison ofthe threshold values to the network conditions or the performance of theflow.

The algorithmic framework disclosed herein ensures that each individualservice flow receives the best possible treatment, satisfying variousper-flow specific QoS requirements such as bandwidth and/or delay, whileall the other service flows are equally satisfied via a heuristicnon-greedy approach. Furthermore, the algorithmic framework is designedto account not only for bandwidth sensitive applications, but also fordelay sensitive applications, while accounting for protocol behaviour.

Some embodiments of the algorithm operate at various protocol layers ina communication network, as long as multiple access technologies areintegrated at the respective layer (e.g., MAC, PDCP, IP, TCP). In someembodiments, the algorithmic intelligence is hosted in at least onenetwork controller module, making it compatible with next generationprogrammable networks. Nevertheless, the algorithmic intelligence may besimply hosted in a multi-link scheduler, which is responsible forscheduling service flows across multiple access technologies. In someembodiments, at least one network controller is involved in the pathcomputation based on collected and available information from at leastone network node. The granularity for path selection/reselection enablesintelligent network controlled traffic steering per user/device, perapplication, per service flow and per service sub-flow. Morespecifically, the path selection algorithmic framework allows differentapplications (e.g., video conferencing, chatting applications, InternetGaming, and the like) to or from the same user/device to use differentconnectivity paths, including paths mapped to different accesstechnologies, simultaneously. The path selection algorithmic frameworkestimates the performance impact that the addition of a new service flowwould have on the other established and ongoing service flows, hencedeciding on accepting/rejecting a new service flow. As a result of thealgorithmic intelligence, the most suitable access technology isselected/reselected seamlessly and on a continuous basis (e.g.,frequency of selection/reselection is a parameter of the algorithmicframework).

Embodiments of the path selection algorithm disclosed herein can accountfor several significant challenges faced by conventional communicationsystems. For example, source bit rate may change without informing thenetwork layer of reason. The path selection algorithm framework can bemade aware of such source related fluctuations and separate them outfrom other potential disturbances in the network. For another example,traffic flows often pass through network elements (e.g., routers) whichdo not interact/report with the network controller(s). Some necessarycross-layer information is not available through standard interfaces onaccess technologies (e.g., LTE, Wi-Fi). Conventional user equipment ismodified, e.g., to incorporate the UCM 147, and therefore embodiments ofthe past selection technique disclosed herein are minimal or lightweightto alleviate the dependency on the mobile vendors and privilege anetwork based solution for traffic steering. In some cases, the TCPlayer responds to network conditions independently, causing havoc.Furthermore, some embodiments of the proposed technique take intoaccount information such as knowledge of the end-user devices (e.g.,Operating System (OS), device capabilities, and the like), knowledge ofapplication requirements (e.g., bandwidth, latency, and the like), andknowledge of network conditions (e.g., utilization level per accesstechnology).

FIG. 2 is a block diagram of a second example of a communication system200 that implements intelligent path selection according to someembodiments. The communication system 200 includes base stations 201,202, 203 (collectively referred to herein as “the base stations201-203”) that operate according to different access technologies. Forexample, the base station 201 is a multi-access technology node thatoperates according to 4G, 5G, and Wi-Fi standards, the base station 202operates according to millimeter wave communication, and the basestation 203 supports Wi-Fi standards. The base stations 201-203 providewireless connectivity to one or more user equipment 205. In theillustrated embodiment, the base stations 201-203 provide wirelessconnectivity to a core network 210 via an edge cloud 215. For example,the edge cloud 215 can be implemented as a mobile edge computing (MEC)device.

The user equipment 205 implements a transceiver (or transmitters andreceivers) according to multiple access technologies. For example, theuser equipment 205 implements an LTE transceiver 220, a 5G transceiver225, and a Wi-Fi transceiver 230. The transceivers 220, 225, 230 areable to establish concurrent wireless connections with the correspondingbase stations 201-203. In the illustrated embodiment, an MPM 235 and anMLS 240 are implemented in the edge cloud 215. A UCM 245 is implementedin the user equipment 205, e.g., as software executing on a processorimplemented in the user equipment 205. Some embodiments of thecommunication system 200 are therefore able to implement intelligentpath steering to selectively allocate paths supported by differentaccess technologies to flows or sub-flows between the user equipment 205and the core network 210.

Intelligent traffic steering as implemented in some embodiments of thecommunication system 100 and the communication system 200 utilizeseveral heuristic concepts. A fast path switching algorithm is hosted onMPM/RCM modules, using user throughput, network delay and radiocondition measurements to determine the suitability of an accesstechnology and its performance. The UCM implemented by a user equipmentcan request fast path switching to preserve a minimum QoE level. Thesystem generates a centralized view of the network load and userperformance, which is available at MPM/RCM through a network controller.The UCM supports passive user data downlink throughput measurement andthe RMM, MPM, and RCM support passive uplink throughput measurement.Uplink and downlink packet delay measurements are performed per accesstechnology, per mobile device and per service flow using an activeprobing mechanism through MPM/RCM and UCM. Jitter measurements can alsobe performed. The system supports radio resource utilization monitoringand reporting by RMM to MPM/RCM for available bandwidth and networkcongestion level estimation, radio resource availability estimation andreporting from a particular mobile device to MPM/RCM, if suchinformation is not available from a RMM entity, and packet reorderingfunction at UCM for downlink user data traffic and at MPM/RCM for uplinkuser data traffic.

Some embodiments of path selection performed in the communicationsystems 100, 200 include congestion detection over the connected accesstechnologies. The UCM (for downlink traffic) and MPM/RCM (for uplinktraffic) monitor the service performance for a target Class-i connectionand report to the MPM/RCM whenever the measured performance getsdegraded below a quality threshold. In some cases, values of themeasured performance metrics trigger MPM/RCM to engage in a pathswitching procedure. The RMM modules that are part of the radio basestations (e.g., eNB for LTE and Wi-Fi APs) report their resourceavailability (i.e. congestion information) to the MPM/RCM. For example,an eNB monitors the utilization of LTE channels (e.g., PDSCH, PUSCH,PDCCH and PUCCH) and calculates in turn a weighted average utilization.In some embodiments, if the level of available resources (e.g., thenumber of physical resource blocks, PRBs) falls below a thresholdtarget, the eNB starts reporting the measured level of availableresources to MPM/RCM. The threshold target for starting reporting andthe actual frequency of such reporting depend on the targetedsensitivity and desired agility of the dynamic path reselection.Alternatively, if an eNB does not have such information, the values ofthe performance metrics can be estimated. In some embodiments, if ameasured idle airlink time level at a Wi-Fi access point falls below athreshold time target, the Wi-Fi access point starts reporting themeasured idle airlink time to the MPM/RCM. The threshold time target forstarting reporting and the actual frequency of such reporting depend onthe targeted sensitivity and desired agility of the dynamic pathreselection. Alternatively, if a Wi-Fi access point does not have suchinformation, the performance metrics can be estimated.

FIG. 3 is a block diagram of an NFV architecture 300 according to someembodiments. The NFV architecture 300 is used to implement someembodiments of the communication system 100 shown in FIG. 1 and thecommunication system 200 shown in FIG. 2. The NFV architecture 300includes hardware resources 301 including computing hardware 302,storage hardware 303, and network hardware 304. The computing hardware302 includes one or more processors, the storage hardware 303 includesone or more memories, and the network hardware 304 includes one or moretransceivers, transmitters, receivers, or other interfaces. Avirtualization layer 305 provides an abstract representation of thehardware resources 301. The abstract representation supported by thevirtualization layer 305 can be managed using a virtualizedinfrastructure manager 310, which is part of the NFV management andorchestration (M&O) module 315. Some embodiments of the manager 310 areconfigured to collect and forward performance measurements and eventsthat may occur in the NFV architecture 300. For example, performancemeasurements may be forwarded to an orchestrator (ORCH) 317 implementedin the NFV M&O 315. The hardware resources 301 and the virtualizationlayer 305 implement virtual resources 320 including virtual computingresources 321, virtual storage resources 322, and virtual networkingresources 323.

Virtual networking functions (VNF1, VNF2, VNF3) run over the NFVinfrastructure (e.g., the hardware resources 301) and utilize thevirtual resources 320. For example, the virtual networking functions(VNF1, VNF2, VNF3) may be implemented using virtual machines supportedby the virtual computing resources 321, virtual memory supported by thevirtual storage resources 322, or virtual networks supported by thevirtual network resources 323. Element management systems (EMS1, EMS2,EMS3) are responsible for managing the virtual networking functions(VNF1, VNF2, VNF3). For example, the element management systems (EMS1,EMS2, EMS3) may be responsible for fault and performance management. Insome embodiments, each of the virtual networking functions (VNF1, VNF2,VNF3) is controlled by a corresponding VNF manager 325 that exchangesinformation and coordinates actions with the manager 310 or theorchestrator 317.

The NFV architecture 300 may include an operation support system(OSS)/business support system (BSS) 330. The OSS/BSS 330 deals withnetwork management including fault management using the OSSfunctionality. The OSS/BSS 330 also deals with customer and productmanagement using the BSS functionality. Some embodiments of the NFVarchitecture 300 use a set of descriptors 335 for storing descriptionsof services, virtual network functions, or infrastructure supported bythe NFV architecture 300. Information in the descriptors 335 may beupdated or modified by the NFV M&O 315.

The NFV architecture 300 is used to implement network slices thatprovide control plane functions or user plane functions which areimplemented in some embodiments of the communication system 100 shown inFIG. 1 and the communication system 200 shown in FIG. 2. A network sliceis a complete logical network that provides communication services andnetwork capabilities, which can vary from slice to slice. User equipmentcan concurrently access multiple slices that support multiple serviceflows between a core network and the user equipment. Some embodiments ofuser equipment provide Network Slice Selection Assistance Information(NSSAI) parameters to the network to assist in selection of a sliceinstance for the user equipment. A single NSSAI may lead to theselection of several slices. The NFV architecture 300 can also usedevice capabilities, subscription information and local operatorpolicies to do the selection. An NSSAI is a collection of smallercomponents, Single-NSSAIs (S-NSSAI), which each include a Slice ServiceType (SST) and possibly a Slice Differentiator (SD). Slice service typerefers to an expected network behavior in terms of features and services(e.g., specialized for broadband or massive IoT), while the slicedifferentiator can help selecting among several network slice instancesof the same type, e.g. to isolate traffic related to different servicesinto different slices.

FIG. 4 is a flow diagram of a first portion 400 of a method of selectingan initial path for a new service flow in a communication system thatsupports multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2.

At block 405, a new connection request is triggered upon the arrival ofa new service flow.

At decision block 410, the network controller checks if the pathselection is a policy driven selection. For example, the networkcontroller can determine whether the path selection is controlled by apolicy set by a network operator. If so, the method flows to block 415.If not, the method flows to decision block 420 and the networkcontroller proceeds with selecting the initial path based on QoEcriteria, such as bandwidth or delay requirements for the new serviceflow.

At block 415, the network controller selects the initial path accordingto policies such as service class, service cost, service priority orload balancing. The network policies may be set up by a mobile networkoperator and may be locally configured by a client (e.g., UCM) at amobile device or by the network through Congestion and Admission Control(CAC) for the selected access technology upon the arrival of the newservice flow. This option ensures a faster service flow connection setup time if the targeted access technology is able to provide thenecessary network resource(s) for the incoming service flow.

If the method flows from the decision block 410 to the block 420, thenetwork controller performs QoE based path selection. In the illustratedembodiment, path selection is performed on the basis of eitherthroughput or delay along the selected path. However, some embodimentsof the network controller implement a combined throughput/delay QoEcriterion. Furthermore, other QoE criteria are used to evaluate thequality of a given service flow and added to the decision tree of thealgorithmic logic in some embodiments of the network controller. Thenetwork controller utilizes measurement reports from all availableaccess channels for the new service flow. In the interest of clarity,the method is discussed below in the context of an embodiment in whichan LTE eNB and a Wi-Fi access point support K classes of service flows(e.g., segregated by geographical areas and different trafficcharacteristics). The following parameters are used in this description:

-   -   Thp_i: the minimum throughput rates (e.g., in Mbps) required for        a service flow of class-i.    -   Delay_i: the maximum radio access packet Round Trip Time (RTT)        delay (e.g., in ms) for a service flow of class-i.    -   LTE_spare_resources: total air-link resources (i.e. available        Physical Resource Blocks (PRBs)) not being used, that may be        allocated to a new service flow.    -   Wi-Fi_spare_resources: total air-link resources (i.e. idle-time)        not being used, that may be allocated to a new service flow.    -   LTE_available_bw: this is estimated per UE, based on the UE's        specific radio conditions, which provides an estimate for the        Modulation and Coding Scheme (MCS) to be used.        LTE_spare_resources together with mobile device's specific MCS        determine LTE_available_bw for the respective mobile device.    -   Wi-Fi_available_bw: this is estimated per Wi-Fi client based on        the client's radio condition, which provides an estimate for the        MCS to be used. Wi-Fi_spare_resources together with the client's        specific MCS determine the Wi-Fi_available_bw for the respective        client.    -   LTE_Pkt_delay: this is the measured RTT through the LTE access        technology between the mobile device and the anchor node (e.g.,        anchor GW).    -   Wi-Fi_Pkt_Delay: this is the measured RTT through the Wi-Fi        access technology between the Wi-Fi client and the anchor node        (e.g., anchor GW).

At the decision block 420, the network controller checks if the pathselection decision is a throughput driven selection based on athroughput requirement for the new service flow. If so, the method flowsto block 425. Otherwise, the method flows to decision block 430.

At block 425, the network controller performs path selection based onthroughput measurements (i.e. Thp_i) over the available interfaces.

At decision block 430, the network controller determines whether thepath selection decision is a delay driven selection based on a delayrequirement for the new service flow. If so, the method flows to block435. Otherwise, the method flows to block 405. Some embodiments of thenetwork controller consider other QoE requirements in addition tothroughput and delay. In that case, the portion 400 method includesadditional branches to account for the additional QoE requirements.

At block 435, the network controller performs the path selection basedon delay measurements (i.e. Delay_i) over the available interfaces.

After completing any of blocks 415, 425, 435, the portion 400 of themethod flows to node 1.

FIG. 5 is a flow diagram of a second portion 500 of a method ofselecting an initial path for a new service flow in a communicationsystem that supports multiple access technologies according to someembodiments. The method is implemented in a network controller such assome embodiments of the RCM 140 or the MPM 141 shown in FIG. 1 or theMPM 235 shown in FIG. 2. The second portion 500 of the method begins atnode 1, which links the second portion 500 to the first portion 400 ofthe method shown in FIG. 4.

At block 505, the network controller identifies any access channels thatare available to connect to the mobile device. The available accesschannels are determined based on defined QoE criteria, which can includean offer to QOS per application class such as throughput or packetdelay, jitter, radio channel conditions, numbers of connected devices,numbers of active devices, numbers of active sessions, radio resourceavailability or utilization levels, mobile device mobility, mobiledevice capability, and the like. For example, an access channel isconsidered available if it has sufficient bandwidth and/or acceptablelatency to accommodate the new service flow. In some embodiments, thecondition “and” is used when both bandwidth and latency need to besatisfied, while the condition “or” is used when either the bandwidth orthe latency needs to be satisfied as part of the QoE criteria.

In some embodiments that implement LTE and Wi-Fi access technologies,the network controller assigns the service flow to the LTE network if:(Thp_i<LTE_Available_bw and LTE_Available_bw>Wi-Fi_Available_bw)AND/OR(Delay_i>LTE_Pkt_delay and LTE_Pkt_delay<Wi-Fi_Pkt_delay)Otherwise, the network controller assigns the service flow to the Wi-Finetwork if:(Thp_i<Wi-Fi_Available_bw and Wi-Fi_Available_bw>LTE_Available_bw)AND/OR(Delay_i>Wi-Fi_Pkt_delay and Wi-Fi_Pkt_delay<LTE_Pkt_delay)If the network controller determines that at least one access technologyis available, the method flows to block 510, otherwise the method flowsto block 515.

At block 510, the network controller assigns the new service flow to atleast one access channel that is selected out of the set of availableaccess channels. In some embodiments, the network controller assigns oneor more access channels to the new service flow based on a priority ifmultiple access channels are found available and a priority policy isimplemented among the available access channels with respect to someservices (e.g., Wi-Fi gets selected over LTE for best effort traffic andLTE gets selected over Wi-Fi for interactive streaming video traffic).In some embodiments, the network controller assigns the access channelthat has the most available resources. The method then flows to node 2.

At block 515, the network controller has determined that none of theaccess channels has enough spare resources to support the servicerequirements (Thp_i and/or Delay_i) for the new service flow by itself.However, the new incoming service flow may be supported on a singleaccess channel by taking some bandwidth away from either all or a subsetof other ongoing service flows, as long as the respective QoS/QoE of allthe affected ongoing service flows is not adversely impacted. In theillustrated embodiment, the network controller evaluates the availableaccess channels to find out the available access channel with thelargest bandwidth that can be released from another ongoing service flowwithout violating the QoE requirements of any already established andongoing service flows. In some cases, a single service flow consumes theentire bandwidth of the access channel that has the most sharablebandwidth. In such cases, the access channel does not have spareresources and the initial path selection mechanism described hereinwould appear to be unable to allocate resources for the new incomingservice flow. Nevertheless, the incoming service flow may be assigned tothe access channel by sharing the access channel's resources among allthe service flows carried out on the access channel. For example,protocols such as TCP would naturally converge to such behavior.Furthermore, if the network resources are shared fairly among all theservice flows of the same access channel, then each service flow may endup using an equal amount of network resources.

On this premise, at decision block 520, the network controllerdetermines whether the equally shared network resources are sufficientto meet the service requirements of the already established and ongoingservice flows while providing sufficient quality to the new incomingservice flow as well. If so, the method flows to block 510 and thenetwork controller assigns new incoming service flow to the accesschannel and the method flows to node 2. If the network controllerdetermines that there are insufficient resources to meet the servicerequirements of the ongoing service flows and the new incoming serviceflow, the method flows to decision block 525.

At block 525, the network controller was not able to find sufficientbandwidth in any of the access channels to accommodate the new serviceflow. The network controller therefore attempts to distribute thepackets in the incoming new service flow across multiple access channelsconcurrently by dedicating a fraction of the bandwidth of all or asubset of the access channels to the new service flow. This is referredto herein as path splitting or service flow splitting. The networkcontroller selects a suitable set of access channels such that theaggregate throughput across the access channels meets the targetedthroughput requirement, Thp_i, of the new service flow. Similarly,traffic of the incoming new service flow that is split into the otheraccess channels must not violate the delay requirement, Delay_i, of thenew service flow. In some embodiments, splitting the service flow acrossthe plurality of access channels is performed to meet a targetend-to-end packet delay, a jitter requirement of the service flow, or acombination thereof.

At decision block 530, the network controller determines whether a setof access channels are available to support the QoE requirements of thenew service flow. If so, the method flows to block 510 and the newincoming service flow is assigned to the set of access channels. If not,the method flows to block 535.

At block 535, the network controller has determined that there isneither a single access channel nor a set of access channels that can beassigned to the new service flow to meet the quality requirements of thenew service flow. The network controller therefore rejects theconnection request for the new service flow.

FIG. 6 is a flow diagram of a third portion 600 of a method ofperformance monitoring and dynamic path switching in a communicationsystem that supports multiple access technologies according to someembodiments. The method is implemented in a network controller such assome embodiments of the RCM 140 or the MPM 141 shown in FIG. 1 or theMPM 235 shown in FIG. 2. The third portion 600 of the method begins atnode 2, which links the third portion 600 to the second portion 500 ofthe method shown in FIG. 5. Initial path selection has therefore beenperformed by the network controller prior to beginning the method 600,e.g. according to the method illustrated in FIGS. 4 and 5.

At block 605, the network controller monitors the performance of aservice flow. Monitoring performance of the service flow includesreceiving feedback from network entities that includes measurements ofperformance metrics performed by the network entities, such as UCM, MLS,RCM, MPM, and the like. For example, once a new service flow is assignedeither to a single access channel or to multiple access channelsaccording to the method illustrated in FIGS. 4 and 5, the new serviceflow is continuously monitored for its performance. Examples ofperformance metrics that are measured include downlink throughputs,uplink throughputs, packet delays, available bandwidth and radioconditions, and the like.

At block 610, the network controller monitors network conditions, e.g.,using information collected by network entities such as UCM, MLS, RCM,MPM, and the like. Examples of the information indicative of networkconditions include an offered QoS per application/service class (e.g.,throughput/packet delay), radio channel conditions, a number ofconnected devices, a number of active sessions, radio resourceavailability and utilization levels, mobile device mobility and mobiledevice capability, and the like. If the monitored performance or networkconditions deteriorate for an established service flow, the networkcontroller triggers a path switching procedure to avoid violating theQoS requirements of the affected service flows.

At decision block 615, the network controller determines whether a pathreselection or path switching procedure should be triggered for anestablished service flow due to the performance degradation of theestablished service flow. Some embodiments of the network controllerdetermine whether to perform path reselection or path switching bycomparing one or more quality metrics (e.g., measured throughput and/ormeasured intra-packet delay of the service flow) to a reference qualitythreshold. If the network controller determines that path reselection orpath switching is to be performed, the method 600 flows to node 3, whichlinks the method 600 to the portion 400 of the method illustrated inFIG. 4. If the network controller determines that path reselection orpath switching is not to be performed, the method 600 flows to decisionblock 620.

At decision block 620, the network controller determines whether pathconsolidation should be performed. For example, in addition to theservice performance degradation considerations, the frequency ofoccurrence of service flow splitting across multiple access channels canbe used to trigger the procedure for path reselection with the goal ofconsolidating and aggregating the split flows back into a single serviceflow. Path consolidation therefore reduces the number of access channelsserving a single service flow. If the network controller determines thatone or more split service flows are to be consolidated, the method 600flows to node 3. Otherwise, the method 600 flows to block 605 andcontinues performance monitoring.

FIG. 7 is a flow diagram of a first portion 700 of a method ofdynamically modifying criteria for performing path reselection or pathconsolidation in a communication system that supports multiple pathsusing multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2. In the illustrated embodiment, the criteria for performing pathreselection or path consolidation are represented as a throughputthreshold. However, some embodiments of the method are used todynamically modify other criteria such as packet delay thresholds,jitter thresholds, combinations of the thresholds, and the like. A firstflow has been previously connected to the communication system, e.g.,according to the portions 400, 500 of the method shown in FIGS. 4 and 5.The flow is therefore carried via one or more access channels. Someembodiments of the method are performed concurrently with the method 600shown in FIG. 6.

At block 705, variables used for the dynamic modification of the pathmodification criteria are initialized. The method utilizes the variablet_cur to keep track of the current system time throughout the entiredevelopment cycle. Another time variable called t_ref is used toestablish time references whenever necessary. The network controllersets t_ref=t_cur during initialization.

In some embodiments, a number of measurement samples are taken during awarmup measurement interval referred to as warmup_interval, which is onthe order of a few seconds in some cases. During this warmup interval, anumber of measurement samples are gathered, where each measurementsample is taken over a warmup (or ramping up) measurement samplinginterval denoted by ΔT1, which can be the same or different from theregular measurement sampling interval, which is denoted by ΔT. As areference example, ΔT1=dT=500 ms.

The existing flow is associated with quality requirementcharacteristics, such as a minimum required throughput target defined as“min_Thp_Thresh_Qual”. The variable “min_Thp_Thresh_Temp” is used tokeep track of the measured throughput for the flow and is initialized to“min_Thp_Thresh_Qual”, i.e. min_Thp_Thresh_Temp=min_Thp_Thresh_Qual.Furthermore, the variable “min_Thp_Thresh_Temp” is dynamically adjustedover time and used to trigger path switching (or path consolidation)events whenever it is deemed necessary, as discussed herein.

The variable “nbr_Temp_Iterations” is used to keep track of countingevents and is initialized to “nbr_Iterations”, i.e.nbr_Temp_Iterations=nbr_Iterations, where the parameter “nbr_Iterations”is set to a constant value. For example, the network controller can set“nbr_Iterations” to the value resulting from dividing the time alarminterval (e.g., 3 sec) by the regular measurement sampling interval ΔT(e.g., 500 ms), which is equal to 6 in this example. This value is usedto trigger a reload of initial values of some control variablesintroduced at this step, after a subsequent dynamic threshold adaptationphase, and to determine whether the connection monitoring was either ina normal phase or in a dynamic threshold adaptation phase.

The variable “var_Count” is used to keep track of throughput measurementsamples within the targeted time alarm interval. Each throughputmeasurement sample is taken over a specific measurement time granularityΔT1 or dT (e.g., 500 ms) and triggers a counting event which incrementsthe “var_Count”. In the illustrated embodiment, path switching (or pathconsolidation) is triggered when “var_Count” is equal to“nbr_Iterations”. The network controller initializes var_Count to 0,i.e. var_Count=0, and a temporary counting variable i is set to 1, i.e.i=1, during initialization.

In order to smooth out the throughput measurements, some embodiments ofthe network controller utilize a weighted average throughput todetermine whether to perform path switching or consolidation. To assistthe weighted throughput computation and the throughput comparisondecisions, the network controller stores the latest set of measurementsamples of size L into an array “weighted_Avg_Thrp[ ]” of size L. Thisis referred by the parameter thrp_Avg_Length, hence_thrp_Avg_Length=L,where L=6 as a reference example. The value of the latest throughputmeasurement sample is stored in the variable weighted_Avg_Thrp[L], thesecond to last is stored in the variable weighted_Avg_Thrp[L−1], and soon, until the (L−1)-th to last is stored in the variableweighted_Avg_Thrp[1]. Once initialization is complete, all the elementsof the array are set to 0, i.e. weighted_Avg_Thrp[j]=0 for any integer jbetween 1 and L. The method then flows to decision block 710.

At decision block 710, the network controller monitors the current timet_cur with respect to the reference time t_ref, which was initialized inblock 705. If the difference between the two is lower than the thresholdΔT1, the method flows to block 715, otherwise the method flows todecision block 720.

At block 715, the network controller obtains the throughput measurementsample thrp_i over the current i-th measurement interval of durationΔT1. At the end of the measurement interval ΔT1, the method flows backto block 710 and a new time condition is tested.

At block 720, the network controller determines whether the currentvalue of the temporary counting variable i is higher than (T1/ΔT1−L).This comparison indicates whether the method is within the last Literations before the end of the predetermined ramping up time intervalT1. If so, the method flows to block 725, otherwise the method flows todecision block 730.

At block 725, the network controller sets the values of theweighted_Avg_Thrp[ ] array of L elements, where k is a temporary integervariable which takes values between 1 and L and represents the index ofthe elements in the array. The first value in the array corresponding tok=1 is set to the current throughput measurement sample thrp_i, i.e.weighted_Avg_Thrp[1]=thrp_i. The other values of the weighted_Avg_Thrp[] array are weighted between the current measurement sample thrp_i andthe previous weighted value, i.e.weighted_Avg_Thrp[k]=A*thrp_i+B*weighted_Avg_Thrp[k−i] for any integer kbetween 2 and L. The parameter A and B determine the relative weights ofthe current and previous measurement samples and are set so that A+B=1.For instance, if A=1 and B=0, the weighted average is memoryless and isdefined as the latest measurement sample. In another example, ifA=B=0.5, the current and previous measurement samples have equal weightsin determining the weighted average value. After completing these steps,the method flows to the decision block 730.

At decision block 730, the network controller determines whether thevalue of the temporary counting variable i is still smaller than T1/ΔT1,which represents the number of throughput measurement samples that areobtainable within the warmup measurement sampling interval T1. If so,the method flows to block 735, otherwise the method flows to block 740.

At block 735, the network controller increments the temporary countingvariable i, i.e. i=i+1, and the reference time variable t_ref isincremented by the warmup measurement sampling interval ΔT1,t_ref=t_ref+ΔT1, to start a new measuring sampling interval.

At block 740, the network controller updates the current weightedaverage throughput with the most recent value obtained at the latestiteration step i, i.e. weighted_Avg_Thrp_Cur=weighted_Avg_Thrp[i], andthen the method flows to the node 4.

FIG. 8 is a flow diagram of a second portion 800 of a method ofdynamically modifying criteria for performing path reselection or pathconsolidation in a communication system that supports multiple pathsusing multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2. Node 4 provides one entry point to the second portion 800 of themethod. The second portion 800 of the method is therefore linked to thefirst portion of the method 700 shown in FIG. 7 via the node 4. Thenodes 6, 8, 9, 10, 11, and 13 also provide entry points to the secondportion 800 of the method. The second portion 800 of the method istherefore linked to a third portion 900 of the method illustrated inFIG. 9, a fourth portion 1000 of the method illustrated in FIG. 10, anda fifth portion 1100 of the method illustrated in FIG. 11.

At block 805, the network controller stores the current time t_cur inthe reference time variable t_ref, so that t_ref=t_cur. Setting thecurrent time equal to the reference time marks the beginning of a newregular measuring sampling interval ΔT,

At decision block 810, the network controller monitors the current timet_cur and compares it against the reference time t_ref. If thedifference between the current and the reference time is smaller thanthe regular measurement sampling interval ΔT, so that t_cur−t_ref<ΔT,the method flows to block 815, otherwise the method flows to block 820.

At block 815, the network controller obtains the throughput measurementsample thrp_ΔT over the current measurement sampling interval ofduration ΔT. At the end of the measurement sampling interval ΔT, themethod flows back to the decision block 810 and a new time condition istested.

At block 820, the network controller updates the weighted_Avg_Thrp arrayby sliding the whole array by one position to capture the latestmeasurement sample. In the illustrated embodiment, for any value of theindex k of the array between 1 and L−1, the value of the elementweighted_Avg_Thrp[k] is overwritten by the value of the elementweighted_Avg_Thrp[k+1]. This is done to ensure proper overwriting acrossthe elements of the array. The latest weighted throughput value of thearray weighted_Avg_Thrp[L] is updated according to a preferred weightingfunction to account for the latest throughput measurement sample thrp_ΔTover the latest regular measurement sampling interval ΔT, i.e.weighted_Avg_Thrp[L]=A*thrp_ΔT+B*weighted_Avg_Thrp[L−1]. As introducedbefore, the parameters A and B determine the relative weights of thecurrent and previous measurement samples, and are set so that A+B=1.Finally, the network controller updates the current weighted averagethroughput with the latest weighted throughput value of the arrayweighted_Avg_Thrp[L], i.e. weighted_Avg_Thrp_Cur=weighted_Avg_Thrp[L].After completing the operations at block 820, the network controllerreaches a branch. On one hand, the method flows to block 805 and beginsa new regular measuring sampling interval ΔT. On the other hand, themethod flows to node 5. The network controller can therefore beexecuting different portions of the method concurrently.

FIG. 9 is a flow diagram of a third portion 900 of a method ofdynamically modifying criteria for performing path reselection or pathconsolidation in a communication system that supports multiple pathsusing multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2. Node 5 provides one entry point to the second portion 800 of themethod. The third portion 900 of the method is therefore linked to thesecond portion 800 of the method shown in FIG. 8 via the node 5. Thenodes 14 and 15 also provide entry points to the third portion 900 ofthe method, which is therefore linked to the fifth portion 1100 of themethod shown in FIG. 11.

At block 905, the network controller compares the currentmin_Thrp_Thresh_Temp against the minimum required throughput targetmin_Thrp_Thresh_Qual for the current service flow. Ifmin_Thrp_Thresh_Temp is smaller than min_Thrp_Thresh_Qual, the networkcontroller determines that the current measured throughput for theservice flow does not meet the minimum quality requirements and themethod flows to node 12. If the network controller determines thatmin_Thrp_Thresh_Temp is not smaller than min_Thrp_Thresh_Qual, themethod flows to decision block 910.

At block 915, the network controller resets the values for the variablesmin_Thp_Thresh_Temp and nbr_Temp_Iterations, with the same setting thatthe network controller used for initialization at block 705 of FIG. 7,i.e. min_Thp_Thresh_Temp=min_Thp_Thresh_Qual andnbr_Temp_Iterations=nbr_Iterations. This indicates that the temporaryquality threshold for the current service flow was not reduced comparedto the minimum required throughput target, and method flows to decisionblock 910. This marks the end of the dynamic threshold adaptation phase.

At decision block 910, the network controller determines whether thecurrent value of the weighted_Avg_Thrp_Cur is belowmin_Thrp_Thresh_Temp, i.e. whetherweighted_Avg_Thrp_Cur<min_Thrp_Thresh_Temp. If this is the case, thenetwork controller determines that the latest weighted throughputmeasurement sample does not satisfy the quality criteria for the currentservice flow and the method flows to block 920, otherwise the methodflows to block 925.

At block 920, the network controller has determined that there are noquality concerns for the current service flow at this point in time.Therefore, the network controller resets the variable var_Count, i.e.var_Counter=0. The method then flows to block 805 shown in FIG. 8 viathe node 6.

At block 925, the network controller has determined that the latestweighted throughput measurement sample is below the quality thresholdfor the current service flow. The network controller thereforeincrements the variable that keeps track of the occurrences of suchquality violations, var_Counter, i.e. the network controller incrementsvar_Counter=var_Counter+1. The method then flows to node 7.

FIG. 10 is a flow diagram of a fourth portion 1000 of a method ofdynamically modifying criteria for performing path reselection or pathconsolidation in a communication system that supports multiple pathsusing multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2. Node 7 provides one entry point to the fourth portion 1000 ofthe method. The fourth portion 1000 of the method is therefore linked tothe third portion 900 of the method shown in FIG. 9 via the node 7.

At block 1005, the network controller determines whether the currentnumber of throughput measurement violations, which is stored in thevariable var_Count, is lower than the current number of iterationsallowed to keep the service flow on the access channel(s) the serviceflow is currently attached to, i.e. var_Counter<nbr_Temp_Iterations. Ifthis is the case, the method flows to block 805 shown in FIG. 8 via node8. Otherwise, the method flows to decision block 1010.

At block 1010, the network controller has determined that all themeasurement samples yield unsatisfactory results. Therefore, the networkcontroller determines whether other access channels are available toserve the reference service flow. If so, the method flows to block 1015.Otherwise, the method flows to block 805 shown in FIG. 8 via node 9.

At block 1015, the network controller selects the best available accesschannel. Some embodiments of the network controller select the accesschannel with the highest estimated spared bandwidth,new_Ch_Est_Spare_BW, as the best available channel. The method thenflows to decision block 1020.

At decision block 1020, the network controller determines whether thelatest weighted throughput measurement sample, weighted_Avg_Thrp_Cur, islower than the estimated spared bandwidth of the best available accesschannel, new_Ch_Est_Spare_BW. If so, the network controller determinesthat there is sufficient bandwidth on the newly found access channel toaccommodate the reference service flow while fulfilling its QoSrequirements and the method flows to block 1025. However, if the networkcontroller determines that there is no alternate access channel toswitch to, the network controller keeps the flow on the current accesschannel and the method flows to block 805 shown in FIG. 8 via node 10.

At block 1025, the network controller switches or redirects thereference service flow to the newly found available access channel. Thenetwork controller also performs the following operations:min_Thrp_Thresh_Temp=weighted_Avg_Thrp_Cur;dynamic_Counter=0;nbr_Temp_Iterations=nbr_Temp_Iterations*SF;where SF is a sensitivity factor which may either increase or decreasethe targeted alarm duration during the dynamic threshold adaptation. Theadjusted values of these variables are used during the dynamic thresholdadaptation. The method then flows to block 805 shown in FIG. 8 via node11.

FIG. 11 is a flow diagram of a fifth portion 1100 of a method ofdynamically modifying criteria for performing path reselection or pathconsolidation in a communication system that supports multiple pathsusing multiple access technologies according to some embodiments. Themethod is implemented in a network controller such as some embodimentsof the RCM 140 or the MPM 141 shown in FIG. 1 or the MPM 235 shown inFIG. 2. Node 12 provides one entry point to the fifth portion 1100 ofthe method. The fifth portion 1100 of the method is therefore linked tothe third portion 900 of the method shown in FIG. 9 via the node 12.

At block 1105, the network controller determines whether the currentvalue of the dynamic Counter variable is lower than thenbr_Temp_Iterations, i.e. dynamic_Counter<nbr_Temp_Iterations. If so,the method flows to block 1110, otherwise the method flows to block1115.

At block 1110, the network controller resets the value of themin_Thrp_Thresh_Temp to the minimum value between the currentmin_Thrp_Thresh_Qual and weighted_Avg_Thrp_Cur, i.e. the networkcontroller sets min_Thrp_Thresh_Temp=min(min_Thrp_Thresh_Qual,weighted_Avg_Thrp_Cur). The network controller then increments thedynamic_Counter, i.e. the network controller setsdynamic_Counter=dynamic_Counter+1. The method then flows back to block805 shown in FIG. 8 via the node 13.

At block 1115, the network controller updates threshold values. In theillustrated embodiment, the network controller computes the lowest valueout of the whole weighted_Avg_Thrp[ ] array, which is stored in thevariable lowest_Temp_Thrp. Then, if the variable lowest_Temp_Thrp ishigher than min_Thrp_Thresh_Temp, the min_Thrp_Thresh_Temp is updatedwith the value of lowest_Temp_Thrp. i.e.min_Thrp_Thresh_Temp=lowest_Temp_Thrp. The method then flows to decisionblock 1120.

At decision block 1120, the network controller determines whether thedynamic threshold adaptation phase is coming to its end by comparingmin_Thrp_Thresh_Temp against min_Thrp_Thresh_Qual. If the networkcontroller determines that min_Thrp_Thresh_Temp is smaller thanmin_Thrp_Thresh_Qual, the method stays in the dynamic thresholdadaptation phase and flows to decision block 910 shown in FIG. 9 via thenode 15. Otherwise, the network controller terminates the dynamicthreshold adaptation phase and the method flows to block 915 in FIG. 9via the node 14.

Embodiments of the intelligent traffic steering techniques disclosedherein have a number of advantages over conventional techniques forutilizing multiple access technologies and communication systems.Simulations of the intelligent traffic steering technique havedemonstrated improvements in reliability and throughput using realistictraffic models for FTP and Internet gaming applications using twoavailable access technologies: LTE and Wi-Fi. The simulation makes thefollowing assumptions:

-   -   The LTE access channel has a 20 MHz bandwidth, while the Wi-Fi        access channel has a 40 MHz bandwidth.    -   There are 14 UEs (mobile devices) which are doing FTP downloads,        where the file size is 20 MB and the FTP traffic model follows a        session inter-arrival time distribution which is Uniform [15        sec, 25 sec].    -   There are 6 UEs (mobile devices) which perform Internet Gaming,        where the average traffic demand for the uplink direction is        sized to 2.5 Mbps and for the downlink is sized to 3 Mbps.        Under these assumptions, traffic steering dynamically        distributes the FTP and gaming traffic over LTE and Wi-Fi.

FIG. 12 illustrates results 1200 that indicate numbers of active userequipment that are connected to LTE or Wi-Fi according to someembodiments. The vertical axis indicates a number of connected userequipment and the horizontal axis indicates time increasing from left toright. The results 1200 demonstrate that when traffic steering isenabled according to embodiments of the techniques disclosed herein,traffic is dynamically distributed between LTE and Wi-Fi. For example,with traffic steering enabled, the initial distribution of userequipment is concentrated on Wi-Fi. As time increases, the distributionof user equipment becomes more balanced and nearly equal numbers of userequipment are connected to LTE and Wi-Fi.

FIG. 13 illustrates results 1300 that demonstrate reliability ofintelligent traffic steering according to some embodiments. The verticalaxis indicates a number of connected user equipment and the horizontalaxis indicates time increasing from left to right. The LTE network failsat approximately five minutes and then recovers less than a minutelater. If intelligent traffic steering is not enabled, the number ofuser equipment connected to Wi-Fi remains approximately the same duringthe failure interval, which indicates that user equipment attached tothe LTE network are dropped and lose their connections. However, ifintelligent traffic steering is enabled, LTE traffic is switched toWi-Fi during the failure time interval. The traffic is switched back toLTE when LTE recovers from the failure. Thus, a network that implementsembodiments of intelligent traffic steering as discussed herein is morereliable than a traffic that does not implement intelligent trafficsteering.

FIG. 14 illustrates a cumulative distribution function (CDF) 1400 of thethroughput for an FTP download according to some embodiments. Thevertical axis indicates the CDF and the horizontal axis indicates theFTP download throughput in Mbps. The entire load is initially equallydistributed between LTE and Wi-Fi. When traffic steering is enabled, thetraffic steering threshold is set to 6 Mbps, which is reflected in theCDF 1400. For example, when traffic steering is enabled, the entirethroughput distribution is above the switching threshold. Enablingintelligent traffic steering according to embodiments of the techniquesdisclosed herein therefore results in a significant difference inminimum throughput: 2 Mbps when traffic steering is not enabled and 8Mbps when traffic steering is enabled.

FIG. 15 illustrates a CDF 1500 of the download time for an FTP downloadwith and without intelligent traffic steering according to someembodiments. The vertical axis indicates the CDF and the horizontal axisindicates the FTP download time in second. Intelligent traffic steeringreduces the average FTP download time by a factor of two compared to theaverage FTP download times when traffic steering is not enabled.

FIG. 16 illustrates a CDF 1600 of packet delays for an Internet gamingapplication according to some embodiments. The vertical axis indicatesthe CDF and the horizontal axis indicates packet delays in the seconds.Intelligent traffic steering according to some embodiments of thetechniques described herein reduces the median packet delay for Internetgaming applications by 18% relative to the median packet delay whentraffic steering is not enabled.

In some embodiments, certain aspects of the techniques described abovemay implemented by one or more processors of a processing systemexecuting software. The software comprises one or more sets ofexecutable instructions stored or otherwise tangibly embodied on anon-transitory computer readable storage medium. The software caninclude the instructions and certain data that, when executed by the oneor more processors, manipulate the one or more processors to perform oneor more aspects of the techniques described above. The non-transitorycomputer readable storage medium can include, for example, a magnetic oroptical disk storage device, solid state storage devices such as Flashmemory, a cache, random access memory (RAM) or other non-volatile memorydevice or devices, and the like. The executable instructions stored onthe non-transitory computer readable storage medium may be in sourcecode, assembly language code, object code, or other instruction formatthat is interpreted or otherwise executable by one or more processors.

A computer readable storage medium may include any storage medium, orcombination of storage media, accessible by a computer system during useto provide instructions and/or data to the computer system. Such storagemedia can include, but is not limited to, optical media (e.g., compactdisc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media(e.g., floppy disc, magnetic tape, or magnetic hard drive), volatilememory (e.g., random access memory (RAM) or cache), non-volatile memory(e.g., read-only memory (ROM) or Flash memory), ormicroelectromechanical systems (MEMS)-based storage media. The computerreadable storage medium may be embedded in the computing system (e.g.,system RAM or ROM), fixedly attached to the computing system (e.g., amagnetic hard drive), removably attached to the computing system (e.g.,an optical disc or Universal Serial Bus (USB)-based Flash memory), orcoupled to the computer system via a wired or wireless network (e.g.,network accessible storage (NAS)).

Note that not all of the activities or elements described above in thegeneral description are required, that a portion of a specific activityor device may not be required, and that one or more further activitiesmay be performed, or elements included, in addition to those described.Still further, the order in which activities are listed are notnecessarily the order in which they are performed. Also, the conceptshave been described with reference to specific embodiments. However, oneof ordinary skill in the art appreciates that various modifications andchanges can be made without departing from the scope of the presentdisclosure as set forth in the claims below. Accordingly, thespecification and figures are to be regarded in an illustrative ratherthan a restrictive sense, and all such modifications are intended to beincluded within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any feature(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature of any or all the claims. Moreover, the particular embodimentsdisclosed above are illustrative only, as the disclosed subject mattermay be modified and practiced in different but equivalent mannersapparent to those skilled in the art having the benefit of the teachingsherein. No limitations are intended to the details of construction ordesign herein shown, other than as described in the claims below. It istherefore evident that the particular embodiments disclosed above may bealtered or modified and all such variations are considered within thescope of the disclosed subject matter. Accordingly, the protectionsought herein is as set forth in the claims below.

What is claimed is:
 1. A method comprising: assigning one of a pluralityof available access channels to a service flow between a communicationdevice and a core network, wherein the plurality of available accesschannels have at least one of sufficient bandwidth or acceptable latencyto accommodate the service flow, and wherein the plurality of availableaccess channels use a plurality of connectivity paths supported by aplurality of access technologies; modifying, based on monitoring atleast one of network conditions or performance of the service flow onthe one of the plurality of available access channels, one of theplurality of connectivity paths that supports the one of the pluralityof available access channels; modifying criteria for modifying the oneof the plurality of connectivity paths for a predetermined time intervalin response to modifying the one of the plurality of connectivity paths;and modifying the one of the plurality of connectivity paths during thepredetermined time interval in response to the modified criteriaindicating that a previous modification of the one of the plurality ofconnectivity paths did not improve a quality-of-experience for thecommunication device.
 2. The method of claim 1, wherein assigning theone of the plurality of available access channels to the service flowcomprises allocating the one of the plurality of connectivity paths tothe service flow based on at least one of an operator policy and aquality-of-service (QOS) requirement for the service flow.
 3. The methodof claim 2, wherein allocating the one of the plurality of connectivitypaths to the service flow comprises allocating the one of the pluralityof connectivity paths to the service flow based on at least one of anoperator preference, a load balancing criterion, a service class, and aservice cost.
 4. The method of claim 2, wherein allocating the one ofthe plurality of connectivity paths to the service flow comprisesallocating the one of the plurality of connectivity paths to the serviceflow based on at least one of bandwidth availability along the one ofthe plurality of connectivity paths, a packet delay along the one of theplurality of connectivity paths, and jitter along the one of theplurality of connectivity paths.
 5. The method of claim 1, whereinassigning the one of the plurality of available access channelscomprises assigning the one of the plurality of available accesschannels based on comparisons of available bandwidth on channels to athroughput threshold.
 6. The method of claim 1, wherein assigning theone of the plurality of available access channels comprises assigningthe one of the plurality of available access channels based oncomparisons of measured packet delays and a delay threshold.
 7. Themethod of claim 1, wherein assigning the one of the plurality ofavailable access channels comprises sharing resources of the one of theplurality of available access channels between the service flow and atleast one other service flow that was previously allocated to the one ofthe plurality of available access channels.
 8. The method of claim 7,wherein sharing the resources of the one of the plurality of availableaccess channels between the service flow and the at least one otherservice flow comprises taking a subset of the resources of the one ofthe plurality of available access channels away from the at least oneother service flow and allocating the subset of the resources to theservice flow in response to determining that a quality-of-service (QOS)or a quality-of-experience (QOE) of the at least one other service flowis not adversely affected by taking the subset of the resources.
 9. Themethod of claim 1, wherein assigning the one of the plurality ofavailable access channels comprises splitting the service flow acrossthe plurality of available access channels in response to insufficientresources being available on any one access channel to support theservice flow.
 10. The method of claim 9, wherein splitting the serviceflow across the plurality of available access channels comprisessplitting the service flow across the plurality of available accesschannels that provide an aggregate throughput that meets a targetthroughput requirement of the service flow.
 11. The method of claim 9,wherein splitting the service flow across the plurality of availableaccess channels comprises splitting the service flow across theplurality of available access channels to meet at least one of a targetend to end packet delay or a jitter requirement of the service flow. 12.The method of claim 1, wherein monitoring the network conditionscomprises monitoring at least one of an offered quality-of-service (QOS)per application class, radio channel conditions, number of connecteddevices, number of active devices, number of active sessions, radioresource availability or utilization levels, mobile device mobility, andmobile device capability.
 13. The method of claim 1, wherein monitoringthe performance comprises monitoring at least one of measurements ofuplink throughput or downlink throughput, measurements of packet delay,measurements of bandwidth availability, and monitored radio conditions.14. The method of claim 1, wherein modifying the one of the plurality ofconnectivity paths comprises at least one of: switching the one of theplurality of connectivity paths to at least one other access channel;and consolidating split portions of the service flow onto a singleaccess channel.
 15. The method of claim 1, wherein modifying thecriteria for modifying the one of the plurality of connectivity pathscomprises modifying threshold values that are used to determine whetherto modify the one of the plurality of connectivity paths based on acomparison of the threshold values to at least one of the networkconditions or the performance of the service flow.
 16. The method ofclaim 15, wherein modifying the one of the plurality of connectivitypaths during the predetermined time interval comprises modifying the oneof the plurality of connectivity paths during the predetermined timeinterval in response to the comparison of the threshold values to atleast one of the network conditions or the performance of the serviceflow indicating that the previous modification of the one of theplurality of connectivity paths did not improve a quality-of-experiencefor the communication device.
 17. An apparatus comprising: at least oneprocessor configured to: assign one of a plurality of available accesschannels to a service flow between a communication device and a corenetwork, wherein the plurality of available access channels have atleast one of sufficient bandwidth or acceptable latency to accommodatethe service flow, and wherein the plurality of available access channelsuse a plurality of connectivity paths supported by a plurality of accesstechnologies; and networking hardware configured to monitor at least oneof network conditions or performance of the service flow on the one ofthe plurality of available access channels, wherein the at least oneprocessor is configured to modify, based on the monitoring of theservice flow, one of the plurality of connectivity paths that supportsthe one of the plurality of available access channels, modify criteriafor modifying the one of the plurality of connectivity paths for apredetermined time interval in response to modifying the one of theplurality of connectivity paths, and modify the one of the plurality ofconnectivity paths during the predetermined time interval in response tothe modified criteria indicating that a previous modification of the oneof the plurality of connectivity paths did not improve aquality-of-experience for the communication device.
 18. The apparatus ofclaim 17, wherein the at least one processor is configured to allocatethe one of the plurality of connectivity paths to the service flow basedon at least one of an operator policy and a quality-of-service (QOS)requirement for the service flow.
 19. The apparatus of claim 18, whereinthe at least one processor is configured to allocate the one of theplurality of connectivity paths to the service flow based on at leastone of an operator preference, a load balancing criterion, a serviceclass, and a service cost.
 20. The apparatus of claim 18, wherein the atleast one processor is configured to allocate the one of the pluralityof connectivity paths to the service flow based on at least one ofbandwidth availability along the one of the plurality of connectivitypaths, a packet delay along the one of the plurality of connectivitypaths, and jitter along the one of the plurality of connectivity paths.21. The apparatus of claim 17, wherein the at least one processor isconfigured to assign the one of the plurality of available accesschannels based on comparisons of available bandwidth on channels to athroughput threshold.
 22. The apparatus of claim 17, wherein the atleast one processor is configured to assign the one of the plurality ofavailable access channels based on comparisons of measured packet delaysand a delay threshold.
 23. The apparatus of claim 17, wherein the atleast one processor is configured to allocate resources of the one ofthe plurality of available access channels for sharing between theservice flow and at least one other service flow that was previouslyallocated to the one of the plurality of available access channels. 24.The apparatus of claim 23, wherein the at least one processor isconfigured to take a subset of the resources of the one of the pluralityof available access channels away from the at least one other serviceflow and allocate the subset of the resources to the service flow inresponse to determining that a quality-of-service (QOS) or aquality-of-experience (QOE) of the at least one other service flow isnot adversely affected by taking the subset of the resources.
 25. Theapparatus of claim 17, wherein the at least one processor is configuredto split the service flow across the plurality of available accesschannels in response to insufficient resources being available on anyone access channel to support the service flow.
 26. The apparatus ofclaim 25, wherein the at least one processor is configured to split theservice flow across the plurality of available access channels thatprovide an aggregate throughput that meets a target throughputrequirement of the service flow.
 27. The apparatus of claim 25, whereinthe at least one processor is configured to split the service flowacross the plurality of available access channels to meet at least oneof a target end to end packet delay or a jitter requirement of theservice flow.
 28. The apparatus of claim 17, wherein the networkinghardware is configured to monitor at least one of an offeredquality-of-service (QOS) per application class, radio channelconditions, number of connected devices, number of active devices,number of active sessions, radio resource availability or utilizationlevels, mobile device mobility, and mobile device capability.
 29. Theapparatus of claim 17, wherein the networking hardware is configured tomonitor at least one of measurements of uplink throughput or downlinkthroughput, measurements of packet delay, measurements of bandwidthavailability, and monitored radio conditions.
 30. The apparatus of claim17, wherein the at least one processor is configured to perform at leastone of switching the one of the plurality of connectivity paths to atleast one other available access channel and consolidating splitportions of the service flow onto a single available access channel. 31.The apparatus of claim 17, wherein the at least one processor isconfigured to modify threshold values that are used to determine whetherto modify the one of the plurality of connectivity paths based on acomparison of the threshold values to at least one of the networkconditions or the performance of the service flow.
 32. The apparatus ofclaim 31, wherein the at least one processor is configured to modify theone of the plurality of connectivity paths during the predetermined timeinterval in response to the comparison of the threshold values to atleast one of the network conditions or the performance of the serviceflow indicating that the previous modification of the one of theplurality of connectivity paths did not improve a quality-of-experiencefor the communication device.
 33. A method comprising: assigning a firstavailable access channel of a plurality of available access channels toa service flow between a communication device and a core network,wherein the plurality of available access channels have at least one ofsufficient bandwidth or acceptable latency to accommodate the serviceflow, and wherein the one of the plurality of available access channelsuses a first connectivity path of a plurality of connectivity pathssupported by a plurality of access technologies; monitoring at least oneof network conditions or performance of the service flow on the one ofthe plurality of available access channels; switching the service flowfrom the first available access channel on the first connectivity pathto a second available access channel of the plurality of availableaccess channels on a second connectivity path of the plurality ofconnectivity paths based on a comparison of a target performance metricto at least one of the network conditions or the performance of theservice flow on the first connectivity path; generating a dynamictemporary performance metric that varies during a ramping up timeinterval in response to switching the service flow from the firstconnectivity path to the second connectivity path; and determiningwhether to switch the service flow from the second connectivity path toa third connectivity path based on a comparison of the dynamic temporaryperformance metric to the network conditions and the performance of theservice flow on the second connectivity path during the ramping up timeinterval.